Summary

Improves GET /annotate/all performance.

Background Context

Issue Police-Data-Accessibility-Project#566 describes improving performance of an endpoint. However, before improving performance I first wanted to measure current performance to drive the decision with data. As such I established a benchmark using pytest-benchmark. To back the benchmark with scale, it seeds 10k URLs to populate some relatively realistic annotation data.

The benchmark determined that main_query_s was by far the largest portion of wall-clock time in this endpoint and so I focused my efforts on improving that performance. I considered a number of different solutions but by far the most effective is to convert the url_annotation_count_view and url_annotation_flags from regular PostgreSQL views to materialized views with unique indexes on url_id.

Performance Improvement

Measured locally with pyinstrument via the new *_profiled benchmark tests:

Staleness Tradeoff

I added the materialized views to the existing RefreshMaterializedViewsOperator schedule which is once per day. In the worst case, URL ranking could be up to 24 hours stale - e.g. a URL that just received its 10th annotation may still be ranked highly when it should be deprioritized. My personal viewpoint is that this is acceptable because:

• No user-facing data is incorrect - only the ordering of work is affected
• No writes or authorization decisions are gated on these values
• Annotators are directed to URLs in a slightly suboptimal order, not a wrong one
• Follows the same trade-off already accepted by the 3 existing materialized views in the codebase

If fresher data is a product priority, I would recommend a more aggressive refresh rate (such as hourly) for the materialized view.

Alternative Long Term Solution

I would be remiss to not include the other, much larger refactor I considered which is to implement proper CQRS. The core of the issue here is that data is prioritized for writes, NOT reads. Then, at query time we pay the price and have to stitch all the data together. Materialized views shift that data stitching slightly left (once a day in a separate job), but a potentially more "complete" fix would be to add a new, fully denormalized table which looks like this:

-- read model, updated on every write command
annotation_queue (
    url_id        PRIMARY KEY,
    priority_rank INT,  -- pre-computed: manual > followed > count > id
    total_count   INT,
    is_manual     BOOL,
    followed_by_user BOOL,
    ...
)

Then at each time we write a new annotation to be consumed by annotators, we also populate this table correctly. Then, at query time we have a primary key lookup (extremely cheap): SELECT * FROM annotation_queue ORDER BY priority_rank LIMIT 1.

If we are interested, I do think this could be a new issue if read performance will continue to be prioritized for the app.

Test plan

• All 4 benchmark tests pass (test_benchmark_annotate_all_*)
• Migration applies cleanly from scratch (alembic upgrade head)
• CONCURRENTLY refresh works (unique indexes in place as prerequisite)